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PREFACE 


The central question of this dissertation is 
whether an objective ranking of utility to the decision 
maker can be assigned to the form of the presentation to 
him. In this instance we are interested in whether an 
executive decision can be reached earlier, faster or more 
consistently with a computer-driven display device than with 
the more customary printed material. 

An experimental group of eighteen management-level 
subjects with extensive experience in inventory control 
was assembled for a two-week short course in advanced 
inventory management techniques. During the short course, 
the subjects were exposed to simulated results from 
computer application of certain inventory control policies 
to a hypothetical inventory system handling n items. This 
system is faced with a certain randomly derived set of 
orders, price changes, replenishments of stock and other 
transactions. The results of the simulation were presented 
on both printer paper and on a cathode ray tube display 
device. For each method of presentation, results were 
represented in both tabular and graphical forn. 

When a substantial part of the course had tran- 
spired, the subjects were asked to evaluate the results of 


simulating two inventory systems, using printer output for 





iv 
one evaluation and the cathode ray tube display device for 
the other evaluation. By means of a Latin square design 
and rank order statistics these evaluations were inspected 
to determine if experienced decision-makers using a display 
device could reach a decision that was earlier, faster or 
more consistent than a decision reached by means of printer 
output. 

The results indicated, with very high statistical 
Significance, that a decision could be made faster with a 
display device, and with high significance that a decision 
could be made on the basis of less information by 
means of the display device. Other results pointed to 
decisions that were more consistent for the individual and 
which tended more to agree with the rest of the group when 
display device techniques were used for the evaluation of 


the inventory systems. 
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Pee NTRODUCTZeH 


The central question of this dissertation sis 
whether an objective ranking of utility to the decision 
maker can be assigned to the form of the presentation to 
him. In this instance we are comparing the utility of a 
cathode ray tube display to printed material, and are 
interested in whether an executive decision can be reached 
earlier, faster, or more consistently with the display 
device than with the more customary printed material. 

In order to provide a vehicle for the experiment, 
and to establish an environment for a discussion of the 
results, a simulator was written for computer application 
of certain inventory control policies to a hypothetical 
inventory system handling n line items. This system is 
faced with a certain randomly derived set of orders, price 
changes, replenishments of stock, and other transactions. 
This simulator is governed by a number of pseudo-random 
variate generators and by program constants. These para- 
meters are set by the user. An important set of these 
parameters establishes the management policy for inventory 
eonbrol. A policy jis formalized iy othesoe lection ms 
combination of three policy components from among: 

l. Two service rules to be used in variable safety 


level computation. 





2e Two smoothinz constants to be used in demand 

forecasting by Exponent lal smoocgnine, 

3. Ihree purchase quantity "Compuvration rules. 

A Cartesian product of these three sets gives a total of 
twelve inventory control policies to be investigated. The 
Simulator exercises each of the twelve policies for a 
period of twenty-four months of generavedw transactions. 

At the end of each of the simulation months, relevant 
statistics are gathered about the system performance under 
each policy set. In order to insure uniform interpretation 
of the statistics, each policy set is exercised under 
Hdentical initial conditions. 

Due to the large number of parameters which are 
available to the user, the simulation program provides a 
versatile vehicle for this investigation. The statistics 
which the program develops are quite extensive, and only 
part of these data are used in the investigation. Although 
the simulation program and its results are of interest in 
their own right, the present investigation uses the simu- 
lator as a vehicle rather than the object of investigation. 

Based on the data presented to him by the simulation 
program, the decision maker is expected to make a series of 
decisions. At the end of each simulation month, he ranks 
the policies in the order of acceptability to him. Each 
decision maker thus produces a set of twenty-four ordered 
lists of twelve policies each, from which conclusions will 


be drawn concerning the form of presentation of the data. 





The experiment was conducted in the context of a 
short course in advanced inventory control techniques. 
A class of eighteen interested and experienced people in 
theminventory control fiecldymet formeatwo weekomauming 
Aucust 1958, for twenty class hours of instruction under 
the auspices of the Department of Computer and Information 
Science at the University of North Carolina in Chapel Hill. 
During the course the following topics, among others, were 
treated: 

1. Exponential and weighted time series smoothing 

for demand forecasting, 

2. Probability distributions of demand. 

3. Svaluation of order-quantity computations. 

4, Variable safety level concepts. 

5. Performance budseting. 

6. Master record file design, 
ijear the end of the course, each participant was given, as 
examination problems, the statistics resulting from two 
different simulation runs. The participants were asked to 
rank the various management policies, month by month and 
to make other decisions regardins the policies, on the 
basis of the data presented. Each participant acting in 
tae capacity of decision*maker, or deetdcr, Wisin beva 
printer output on one problem and graphic display presen- 
tation on the other, demonstrated his decision making 
eae CY 


The statistics from the simulation program were 





disolayed ina variety of formats on standard computer 
orinter paper. The statistics included such data as 
percent availability of stock, number of purchase orders 
renerated, lost sales and total dollar investment in 
inventory. The output was in the @orm of tabular listines 
and bar ztraphs. Also, for each problem, the IBM 2250 
cathode ray tube display unit was prozszrammed to show the 
Same data aS appear on the printer output, in fundamentally 
the same listing and sraphical formats. Thus the exper- 
iment did not attempt to establish the relative effective- 
ness of listed versus plotted data. Instead it investi- 
zated the relative effectiveness of printed versus display 
presentation of such data. The display unit was under the 
control of the decider, who was able to specify which of 
the twenty-four months and the title of the displayed data 
which he wanted to see. In both display and printer output 
cases, he was expected to start at the first month and 
proceed through the months in order, 

The course participants were divided into two test 
sroups, each group consisting of nine individuals chosen 
at random. Tests were made in regard to: 

1. Are decisions made faster? The elapsed clock 
time to decide on a ranking of policies which 
the decider would be Willain> temcommit hina 
to for™fuvure acvUlon was tecuca. 

2. Are decisions made earlier? The calendar month 


(out of the twenty-four months of simulation) 
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an which the decider feels he has enough data 
tO commit himselit CO .a ranking fer tubure scored 
was tested. 

3. Are decisions made faster? The elapsed clock 
time to complete the remainder of the problem 
was tested. After committing himself toa 
ranging gepne parvilcl pant@eonvetmnucs Giles proplen 
through the twenty-fourth month. 

4, Are decisions made more consistently? Individ- 
val correlation coefficients from the Spearman 
rank correlation test were computed, 

5. To what degree do the members of each group 
azree amons themselves in regard to the 
rankings? Kendall's W coefficient of concord- 
ance was computed for each cell of the Latin 
souare. 

In summary, the decider makes two basic kinds of 
decisions. He decides on a ranking at each month in turn, 
and he decides whether or not to indicate at this month 
that he feels confident enoush of his ranking to predict 
that it will not significantly chanse over the remainder 
of tne twenty-four monvchs. 

The resultinz ordered lists of policy sets and the 
preparation times were analyzed in two basic ways. First, 
ae2ex 2 Latin square desitn with repeated measures was 
used in an analysis of variance. Rank order statistics 


were then used to test the consistency of each individual 





decision maker and the concordance of a group of decision 
makers. 

The statistical analysis of variance explored the 
differences between graphic and printed data presentation 
(treatment), between the first and second problems done by 
each participant (order), and isolated the differences 
between the performance of Group I and Group II. 

The rank order of the policy sets at the decision 
month and at the end of the twenty-four month simulation 
was compared by the Spearman rank correlation coefficient 
test for each participant. These rank correlation 
coefficients were then arranged in the Latin square and 
an analysis of variance was performed. These tests 
determined whether treatment, order, or group composition 
caused differences in the consistency of a participant's 
final answer, after he had full data available, with that 
at the month of commitment to a policy ranking. A 
Kendall's coefficient of concordance was computed to 
measure how well the rankings produced by each test group 
on each problem at the decision month agreed among 


themselves. 





Ti, TRS SIMULATION MODEL 


This simulation program is a dynamic model of a 
general inventory control situation, and is intended to 
investigate inventory management policies in regard to 
forecasting, purchase quantity computation and service 
level over a range of items whose characteristics are 
defined by the user. The model was programmed in PL/I 
utilizing the IBM System 360 Model 75 at the Triangle 
Universities Computation Center, Research Triangle Park, 
North Carolina, 

The model establishes identical initial condivirons 
for each of the twelve policies and then generates a 
series of actions which represent daily business trans- 
actions under a particular policy. This series of daily 
actions is continued over a simulation cycle of two 
360-day years for each policy in question. Reports are 
generated after each thirty-day month of simulation. 

The Cartesian product of the sets of decision 
rules for service level (two rules), purchase quantity 
computation (three rules), and forecasting methods (two 
rules), gives twelve distinct cycles of 720 simulation 
days each. 

The following are the formal properties of the 


Simulation model. While the program is well documented 


in the program listing, the tabulation below of the more 
important elements defines the model by a categorization 
appropriate to the simulation of economic systems.+ We 
Vite t ine exozecnouicmvar lablessas Those independcene 
parameters which are treated by the system as inputs. 
They will, in general, be stochastic variables which are 
generated internally by the program's random variate 
generators. Status variables will describe the state of 
the system in relation to an appropriate time period, and 
are generally found to be in a feedback loop which inter- 
acts with the exogenous variables of a preceding time 
period. For instance, the on=jhand inventory level, ONH, 
reflects a status at any one point in time, but is 
dependent on the values of several stochastic variables in 
preceding time periods. Endogenous variables will be the 
output result of interactions between exogenous and status 
variables. These are the numerical results which are 
found on the printed and cathode ray tube display output. 
Where status variables describe the system and interact 
with other variables, endogenous variables are collection 
pPeints for output purposes only. and do ner Anveracra ward 
other variables to affect system performance. Identities 
will be either definitions or tautologies which describe 
model components. For example, the status variable ON 


1Thomas H. Naylor, et al., Computer Simulation 
Techniques (New York: John Wiley & Sons, Inc., 1966), 


PDP. 9-20 9 159-173. 





9 
is defined by the identity ONWH = ONH - TRANS + DUIN, where 


the identifier on the left at time t + 1 is replaced by the 
expression on the right evaluated at time t. Parameters 
are those variables which define the inventory control 
policies and the distributions describing the environment 
under investigation, as distinguished from the exogenous 
variables that describe the particular experiences to which 
the policies are subjected. These the investigator chanzes 
in order to model his particular policies and simulated 
environments. In all cases the identifier mentioned is 

the one used in the PL/I program and is cross-listed in the 


Program Attribute Table. 


EXOGENOUS VARIABLES 


Identifier Desicnioilen 
RAND A uniformly distributed random 


variate in the range 0O to .99., 


Y A random variate from one of the 
ten random variate generators. 


A A number in the range one to ten 
which indicates which of the ten 
random variate generators (see 
paze 20) is being called. 


STATUS VARIABLES 


identifier Dés.cr ition 
ONA On handeinventory 17 Uunitoroen 


issue per line item. 


eas Amount due in from purchase 
action initiated by a purchase 
quantity Gempuwtatiom: in units of 
issue per line item. 





CRIT 


gL O2E 8) 


CDF 


AUF 


AND 


NFPRES 


MDEM 


De MD 


TRAWS 


BNPRICS 


10 


Total units sold the previous 
Vearimin UnivSs ©f Issue versie 
it Cl, 


Total units returned by customers 
over the simulation cyele; in 
units of issue per line iten. 


Frequency of satisfied demand 
this year; in number of demands 
satisfied per line iten. 


Reorder point (variable safety 
level); in units of issue per 
line iten, 


Total demand last year; in units 
of issue per line iten, 


Count-down variable which shows 
the number of days until a due-in 
order is expected to be received 
for this stock number: iii 


Forecast monthly demand; in units 
of issue ver line iten, 


Annual usage (sales this year); 
in units of issue per line iten. 


Total demand this year; in units 
of issue per line iten, 


Frequency of demands which could 
not be filledyeaienumber tor 
demands per line item. 


Demand this Momens 1n univs of 
issue per line iten, 


The demand resistered against a 
stock number on any particular 
transaction; in units of issue 
per line iten. 


The number sold to a customer as 
the result of a demand: in units 
of issue ver line item. 


The computed cost price of on- 
hand inventory after receipt of 
due-in merchandise; in dollars 
per unit of issue. 





FCST1 


TRND 


ERROR 


TREND 


BEXPCDMD 


MAD 


MRD 


olevene slic aiisbe 


AVAIL 


AVSINV 


LOSS 


PURCH 


a: 


The mean of the number of units 

returned by a customer arainst a 
stock number; in units of issue 

per line item. 


Forecast demand at previous time 
period; in units of issue per 
line item. 


Demand trend: in units of issue 
per line item. 


Error in forecast of demand 
during last time period; in units 
of issue per line item. 


Smoothed trend forecast; in units 
of issue per line iten, 


Expected demand after trend 
correction to forecast demand; in 
units of issue per line iten. 


liean absolute deviation of true 
demand from expected demand; in 
units of issue per line iten. 


Maximum reasonable demand from 
variable safety level computa- 
tions: in units of issue per line 
item. 


ENDOGENOUS VARIABLES 


Description 


Availability ~tmeeraticowor sums 
of satisfied demand to units of 
total demand; in percent per 

fo dlsiery ar 


Total inventory investment, the 
cost of the on-hand inventory 
plus due-in; in dollars per 
DOLICcy. 


Lost sales, the cost of not 
having stock which was demanded 
over the life of the simulation; 
in dollars per policy. 


Cumulative number of purchase 


actions generated; in number of 
purchase actions per policy. 





YW) 


ie anes 


S_ CLITEM 


S CLITEM 


GY) 


_CLITEH 


S_LNITEM 


S_LNITEM 


S_ LNITEM 


S_LNITEM 


S LITEM 


S_LNITEM 


S LNITEM 


(1) 


(2) 


(3) 


(5) 


(6) 


7.) 


12 


Total of items with unit price 
less than 325.00 and annual 
demand less than 3100.00; in line 
Items per wwelicy. 


Total of items with unit price 
less than $25.00 and annual 
demand > $100.00 and 4 31,000; 
in line items per policy, 


Total of items with unit priced 
$25.00 and annual demand» 
$1,000; in line items per policy. 


Total of items with unit price 
over 525.00; in line items per 
DOlLLeCVs 


Total of items with availability 
<70 percent; in line items per 
policy, 


Total of items with availability 
-~ 70 percent to 475 percomu aan 
line items per policy. 


Total of items with availability 
> 75 percent to < 80 percent; 
in line 1tems per pollive,, 


Total of items with availability 
> 80 percent to 4ee5spercenam rnd 
line items per policy. 


Total of items with availability 


> 85 percent to < 90 percent; in 
line items per policy. 


Total of items with availability 
= 90 percent to 2995 percem: sama 
line items per policy. 


Total of items with availability 
= 95 percent, In dine i Veenar cer 
jeroglibe 


IDENTITIES 


The following identities are expressed in terms of 


a programming language assignment statement, where the 


identifier on the left at time t + 1 is replaced by the 





TRANS 


#SOLD 


BNPRICE 


Pale 


ONH 


DUIN 


lets) 


HQTy 


SOOM 


NEX 


MDEM 


sb. 


TRND 


TREND 


13 


expression on the rignt, evaluated at time ¢, 


DiiD if total demand satisfied, else 
TRANS. = ONH; in units of issue per line 
item, 


Blh 
2. TRANS; in units of issue per line 
i item, 


OWH = FRICKE + Boos DUIN ; wim lance pcr 
ONH + DUIN lines item. 


BNPRICE; in dollars per line iten. 


ONH =- TRANS + DUIN; in units of issue 
per line iten. 


HQTY + (TIME/30) * PRD; in units of 
issue per line item. 


PRD * (1 —- BCONSTANT) + MDEM * 
BCONSTANT; in units of issue per line 
iten,. 


+? * HOMDERC@GST * AUP for EOQ; sin Unie. 
HHOLDCOST 


of issue per line item. (Other 
decision rules for HQTY are outlined 
im tex) 


AND/( FREQ + NFREQ); in units of issue 
per line iten. 


EXPEC/2; in units of issue per line 
item, 


AND/FREQ; in units of issue per line 
iten, 


30 
VEZ. DEMD; Fn uni Us on Mice ies cin mintdic 
is} ee 1h item. 


NEX/10; in units of issue per line 
iten, 


PRD =- FCST1; in units of issue per line 
item. 


(BCONSTANT * TRND) + (1 = BCONSTANT) * 
TREND; in units of issue per line 
Tee ia 





KRROR 


KRAPCDHD 


MAD 


MRD 


Identifier 


pele Oye: 


AIRS 


BCONSTANT 


BSAFE 


BYRC 


BD 


HORDERCOST 


HHOLDCOST 


HMAX 


HMIN 


14 


MDE - 2XPCDMD; in units of issue per 
line aivem, 


Papo fee Diee  e=e PeolSral or) 7 
BCONSTANT; in units of issue ver line 
item, 


ABS (BCONSTANT * ERROR) + (1 - 
BCONSTANT) * MAD; in units of issue per 
line item, 


( (TIME/30) * EXPCDMD) + KFACT * MAD * 
SORT (TIME/30)> im units) oneeocuemoer 
line item. 


PARAMETERS 


Description 


Cost price of inventory item; in 
dollars per unit of issue. 


Purchase lead time; in days per 
purchase order per line item. 


Exponential smoothing constant; 
a decimal fraction. 


Fraction of demand unsatisfied, 
for use in variable safety level 
computation; a decimal fraction, 


Number of years in total time 
period of simulation; in years 
per simulation cycle. 


Number of days per month; a 
numeric constant in the range 
ene CO Chartiae 


Order cost; inf dollars per 
purchase action. 


Holding cost; in dollars per 
year. 


Maximum purchase Quanvivy; an 
units of issue per transacricn, 


Hinimum purchase quantity; in 
units of issue per transaction, 





LS 
SAWNUALSDIMD Annual dollar demand comparand 
for purchase quantity compu- 
takdon;itamdollars pem year. 

In order to provide for future demands against 
inventory, it is usual to use past demand history as a 
basis for forecasting this future demand. In this model, 
exponential smoothingé was used to operate on past and 
present demand to produce a forecast demand for use in 
purchase quantity and safety level computation. Single 
exponential smoothing is defined to be: 

NOP eG Gren GSE 
where Y; is the average computed for the end of time 
period t and x, is the demand registered in time period t. 
The smoothing process can be recursively applied, 
yielding higher degrees of smoothing expressed by: 

ve = Ave? 4 (1- XK) vey 
where k is a superscript which defines the degree of 
smoothing, and t indicates the time period in question. 


Double smoothing, for example, requires the following 


calculations: 
eet 1 
Ye = AX, + eae Ye 
YY = Avs (1-KexZ 
1 eee t t=-l 


While of interest for theoretical purposes, hizsh 


orders of smoothing in general provide little practical 


“Robert G. Brown, Statistical Forecasting for 


Inventory Control (New York: McGraw-Hill Book Company, 
1959), pp. 45-51. 





16 
benefit in terms of forecast accuracy, and obviously 
increase file storage space requirements and computation 
time. For these reasons, only single and double smoothing 
nave been used in the model. 

From the definition, however, it can be seen to be 
a trivial matter in PL/I to prosram a recursive function 
call for any desired degree, assuming the availability of 
k storage locations with each master file stock item for 
4th degree exponential smoothing to hold the k values of 
yk, 

The specification of two alternative values for the 
o&-constant in the forecasting routine is at the option of 
the user. For preliminary investigation, values of 0.2 and 
0.4 are customary, but, of course, any values may be 
specified. The specification of the A-constants completes 
the first of the three sets of parameters needed to iden- 
tify the twelve policies to be simulated. 

The second set of parameters controls the purchase 
quantity computation. Here three choices are available for 
composing the twelve-policy set and the user may either use 
the three decision rules embedded in the program, specify- 
ing their parameters as desired, or entirely new rules may 
be specified in any combination with little additional 
Eat OFC 

When the stock on-hand gets to a certain level 
(CRIT) a computation is made to determine how much to 


purchase to replenish this stock. This aperiodic action 
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was taken in lieu of a periodic reorder cycle on the 
assumption that we are dealing with high demand items 
(as opposed to insurance items) and that if there is new 
fitormation, use should pe made ol Vi tame Uavse nei mines — 
mation is in the form of the variable safety level (CRIT) 
and the total demand forecast. This philosophy could have 
been carried one step further and a new exponentially 
smoothed demand produced at this point, but with monthly 
forecasting in use this was deemed unnecessary. The 
purchase quantity computation methods were chosen in order 
to provide three quite distinguishable computations so as 
to give clear alternatives in the evaluation of the 
results. The purchase quantity computation methods chosen 
to model were: 
1. Standard EOG computation ie tans 

HORDERCOST = Order cost = 370. 

HHOLDCOST = Holding cost = 15 percent of price. 

AUF = Annual demand from master file, 


HAMAX twelve months of forecast 


Maximum buy 
demand, 


HNIN = Minimum buy one month of forecast 


demand, 


2. Modified FOQ computation with different para- 
meters in order to accentuate any distinction 
with the standard HOW computation: 

Be, W.. Plossl and Oj W. Witehnt, Seeeductieneend 


Inventory Control (HZnglewood Cliffs, N. J.: Prentice-Hall, 
Inc., 1967), pp. 388-390, 
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Bre 
membenR COS! = Order cost = 5100. 


HMAX = Maximum buy = twenty-four months forecast 
demand. 


HE ORSCOsT = 2oldinewcesy = Me nercent of vricer 

EMIN = Minimum buy = two months forecast demand, 
The decision rule for the modified EOQg computation was to 
use the standard £02 formula with the above parameters if 
the forecast annual demand in dollars was less than 31,000, 
If the annual demand forecast was equal to or greater than 
$1,000, but less than $10,000, the purchase was to be 
Simply twelve months of predicted demand. Otherwise the 
standard HOQ formula was used with the above parameters 
except that HORDERCOST = #3500. 

3. Three way function of annual dollar demand 

homo ea cr! 

1) If forecast dollar demand is less than 
$1,000, buy twenty-four months of forecast 
demand. 

2) If forecast dollar demand is equal to or 
greater than $10,000, buy two months of 
forecast demand. 

3) Otherwise buy six months of forecast demand. 

These three methods represent enough of a variation so 
that the results show significant differences from which 
to choose. Of course any other set of purchase quantity 
computation methods could have been chosen. Since the 


number of simulation cycle runs is dependent on the 


s,s ea ™ 
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ag 

Cartesian product of the overall number of rules in each 
set under consideration, the addition of more rules must 
be considered with total run time in mind. 

The third and last set of rules involves a choice 
of two performance levels or service levels.4 The service 
meoel is mr essence the desived availability Oo: stocker 
the system. For instance, a service level of 90 percent 
indicates that the system economics are expected to be 
able to satisfy 90 percent of the demand against system 
items. The parameter BSAFE may be set to reflect two 
different service levels which are to be investigated. If 
service level is defined as fraction of demand to be 
satisfied from stock, BSAFE will be defined as fraction of 
demand unsatisfied, or fraction of demand back ordered. If 
we are investigating the feasibility of system support for 
service levels of 95 percent and 98 percent, the values of 
the parameter BSAFE would be specified as 0.05 and 0.02, 
The principal use of this parameter is in the computation 


of the variable safety level, CRIT. This safety level, 


or reorder point, varies with each item as the item's 
demand pattern changes. The variable safety level compu- 
tation takes into account past forecast errors and produces 
a small safety level of stock for a well behaved demand 


pattern and a large safety level for a demand pattern 


‘Brown, Statistical Forecasting, pp. 105-115. 


*Ibid., pp. 136-159. 
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which produces large forecast errors. The variable safety 
level computation is accomplished monthly, at forecast 
time, for each™steck item in the file. 

For the ourpeses Of tChisemedel an inventory contre. 
policy was defined to be one of the twelve possible con- 
binations made by selecting one element from each of the 
above three sets of parameters. For example, one policy 
might be X= .15, standard EOQ with parameters as given in 
the model, BSAFE = .02. 

The user has the ability to specify initial stock 
conditions and also the probability distributions for the 
following variables: 

T; Retummref material te sveck, 

2. Frequency of demand. 

3. Price change on receipt of material. 

4, Quantity demanded against individual items. 

In addition, the user may wish to simulate other types of 
transactions than are provided for in the model. The 
program modification in this case would be left to the 
user. To allow flexibility in introducing new subroutines 
to extend or change the model, the following pseudo-random 
variate generators are available as simple PL/I function 
calls: 

» Uniforn, 

. Normal. 


«  Poeisson. 


iS Gy sae 


» Geometric. 





5. Gamma. 
iq © ealoueyonble.T ¢ 
Vey DEG BeCOme Er uc. 
8, Log-Normal. 
9. Negative Binomial. 
10. Exponential, 
The index of the random variate generator in the 
above list corresponds to the value of the exosenous 


variable X (see page9 ). 


Fa 
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III. THE PRESENTATION TO THE DECISION MAKER 


The simulation statistics that are available ona 
monthly basis are shown in Figures 2, 3, and 4& Figure 1 
is a three-dimensional representation of how data are 
collapsed into the matrix in Figure 2. The matrix entries 
correspond to the number of line items which fell into that 
matrix element as of that point in time. The vertical 
columns, or categories, are totaled to show the number of 
line items in each price-demand category. The totals at 
the bottom of the figure are sub-tabulated by the price- 
demand categories. Only the total investment figure is 
scaled. A scale of 100,000 was applied to this due to 
restrictions on the allowable number of characters per line 
on the display surface of the cathode ray tube. A total 
investment of $423,251.81 would, therefore, be reflected 
as $4.23. 

Figure 3 is a tabular monthly recapitulation which 
is ordered by several of the statistics of interest. This 
is the counterpart of Figure 4 which shows one statistic 
across all policies as a bar graph. The printed bar graphs 
are produced monthly for lost sales, purchase order gener- 
ation, total investment and stock availability. 

Once the printer output was specified, it was 


natural to attempt to reproduce this presentation on 
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UNIT PRICE $$25 


<$lOO >$100and >$lI000 
<$ OCo 

SSS 

ANNUAL DOLLAR DEMAND 


Fig. 1.--Three-dimensional representation of Figure 2 





MONTH 16 POLICY 2 PROBLEM @ 13 
OnrT prRIce < $25 nit price > $25 
ANNOAL DOLLAR DEMAND| < £100 1 = £100 & < $1,000 | > £1,000 ' | 
AVAILABILITY (Ree. [cece Dee Se ets (od eR OE ee LINE TOTALS 
\ 0 \ 0 i 6 ; 0 j 0 
< 70% (en oe (eee (| fp 
f a] | 0 I 0 ! 0 ] 0 
> 70€ to < 75% [fee ee ee ee (Jee, «AE nS) ee en 
| 0 ] 0 t 9 | 1 | 1 
> 75% to < BOS eee 5 ee nay |: So a dn ce (a2 ee > |) es ee 
' 0 1 0 I 2 1 0 1 2 
> AOR to < REE Wee a = en ee a ener 2) Se ae eee 
] ] | 9 I ] 3 
> BSE to < 90% Ui a NN Se ay 2 | 
] 0 | 0 1 1 ( 1 ] 2 
> 90€ to < 9EE (ea es 9 {See ere, 26 Rie e ei een a, pete es RE Es | ea Der Oy ene cist = 1 
| 5 1 rf 1 13 1 7. 1 29 
Se GE€ €> © 100% Fille os (<2 die Deere eaten “tr |... ar | _ {aR eee | 
CATEGORY TOTALS: 5 ¥ 16 12 
TOTAL NUMBER OF ITPMS: 37 
t 
CATAGORY LOST SALES AVAILABILITY TOTAL INVPSTMENT 
1 £.00 99.99 $.01 
2 $6.78 99.23 £.02 
3 $16,583.48 93.97 £1.07 
4 $125, 680.56 A3.69 £3.uF 
OVPRALL: $102,070.75 92.90 £4 .S7 
CUMULATIVE NOMBER OP PURCHASE ACTIONS POR THIS POLTCY = 57 
CUMULATIVE NUMBPR OP TRANSACTIONS GPNERATED UNDER THIS POLICY = 1931 


Fie, 2.--Detailed listing of simulation results 
By DOllcy wi Coinmemenrn 
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SORT H __16 PROBLES ¢ 13 











RESOLTS OF _SIAULATION BY i POLICY _FOMBER. 
POLICY #9 RSPR AVAILABILITY TOTAL IR VESTAFRT LOST SAL"S POBCHASE ACTIORS TRABSACTIONS 
1 89.37 $3.80 $222,191.56 106 19397 
2 92.96 $4.57 $182,070.61 57 1931 
3 89.73 $2.73 $50,025.95 67 1948 
a 89.32 $2.78 $153,879.6¢ 73 19.96 
© 92.98 $2,79 $232,559.37? 65 1956 
6 91.061 $3.88 $228,220.50 89 1909 
7 9$.15 $2.77 $23,692.03 73 19 33 
a 96.31 $8.27 $82,522.65 56 19 26 
Q 94.60 $8.16 $185,829.12 98 1961 
10 8b. 6A $2.60 $180,235.37? 71 1949 
WwW 93.78 $5.37 $152,6%6 .06 53 2018 
jap Spell) CER ecn ee Renee eer ears aye, Gino See eee en Seve $2486... 9 I I. ee aaa 
RESOLTS_OF _SISOLATION BY i; TOTAL __JEVESTALET 
POLICY BOABPR AVAILABILITY TOTAL IRVESTAERT ROS Tsao PowCcwase aACTIORS TRANSACTIONS 
10 86.68 $2.60 $108,235.37 71 19869 
3 89.73 $2.73 $50,025.95 67 1968 
7 95.15 $2.77 $23,692.03 73 1933 
a 89.32 $2.78 $153,879.62 73 1896 
12 93.46 $2.66 $6*,199.18 61 19 22 
1 89.37 $3.80 $222,191.56 108 1937 
6 91.01 $3.88 $228,220.50 69 1909 
“J 9a. 60 $a, 16 $185,829.12 98 1861 
A 96.31 $a.27 $82,522.65 56 1926 
2 92.90 $e,57 $182,970.81 57 1931 
5 92.98 $a.79 $232,559.37 65 19%6 
3 i ed as we a +1 Lares bb. a is a) ee eer 1) | eee 
RESOLTS OF _SIAOLATION BY: AVAILABILITI. 
POLICY ROABER AVAILABILITY TOTAL IWVESTAPET LOST SAaL"s PORCHASH ACTIONS TRABSACTIONSE 
R 96.31 $a,27 $82,522.65 56 19 26 
7 95.18 $2.77 $23,692.03 73 193) 
9 98.60 $8. 16 $185,829.12 96 1681 
11 93. 78 $$.37 $152,668.06 $3 2018 
12 93.06 $2.66 $65,199.18 61 1922 
5 92.98 $a.79 $232,559.37 65 1956 
2 92.90 $4.57 $182,070.81 5? 19 31 
6 91.01 $3.86 $228,220.50 69 1909 
3 89.73 $2.73 $50,025.95 67? 1948 
1 69.17 $3.80 $222,191.56 108 1937 
8 69.32 $2.78 $153,879.62 73 1896 
aoe eas | | es ee 7 St | een eeeermes fll Ly YS | eee comer 5 Pe ee Yb a ee 
RESULTS OF _SISOLATI ON BY i LOST SALZS COS75__ 
POLICY BTRGPR QVAILABILITY TOTAL ISVESTRERT LOST S&LES POSCRASH ACTIONS TRAUSACTIORNS 
? 95.15 $2.77 $23,692.03 73 1933 
6 96.31 $8.27 $42,522.65 s 19 26 
3 89,73 $2.73 $50,025.95 6? 1968 
12 93.86 $2.86 $65,199.18 61 1922 
2 92.90 $4.57 $182,070.81 5? 1931 
10 66.68 $2.60 $188,235.37 71 1689 
9 98.60 $8. 16 $185,829.12 98 1661 
W 93.76 $5.37 $152,666.06 $3 20 18 
a 69. 32 $2.78 $153,879.62 73 1896 
1 69.37 $3.80 $222,191.56 108 19 37 
6 91.01 $3.88 $2 28,220.50 09 1909 


a nee 1S | See) | enn PPL Vs pe a es ee 


Fig. 3.e--Sorted listing of simulation results 
sy policies within month 
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Fig, 4,--Bar graph of simulation results 
by month across all policies 
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another device for the experiment. The experimental 
machine was chosen to be the IBM 2250 Model 1 cathode ray 
tube display device.© See Figure 5. This particular 
device is program controlled from the multiprogramming 
environment of the IBM System 360 Model 40 computer at the 
University of North Carolina Computation Center. A program 
was written to take the data output of the simulation 
program and mimic the printer output formats on the display 
device. Some exploration of the power and utility of such 
a display device was made within the confines of the 
desirability of maintaining similar presentation format for 
experimental design purposes. Figures 6 and 7 are the 
display device representations of Figures 2 and 3 and show 
some rearrangement of material due to display character and 
line number limitations.’ Figure 8 shows the first marked 
departure from the printer output of Figure 4. The layout 
is essentially the same, but the presentation is dynamic. 

A programmed timer was specified in the display program to 
advance all lines of the bar graph simultaneously at one 
second intervals. If the user specifies a month at which 


he wishes to examine the data, and then specifies the bar 


6International Business Machines Corporation, 





Unit Model 1, Form A27-2701-1 (January 27, 1967), pp. 5—-37% 


7International Business Machines Corporation, 
IBM System/360 Operating System, Graphic Programming 


Services for IBM 2250 Display Unit, Form C27-6909- 
December 1967), p. 14. 











Fig. 5.--IBM 2250 cathode ray tube display device 
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CATEGORY TOTALS: 4 16. 1/4 
TOTAL NUMBER OF ITEMS: 


CATEGORY LOST SALES % AVAILABLE TOTAL INVESTMENT 
6 
2 b let 
3 16,583 93 
4 125,489 $8 


OVERALL 142,616 4 


CUMULATIVE NUMBER OF PURCHASE ACTIONS FOR THIS POLICY 





Fig. 6.--Display of Figure 2 
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“ig, 8,--Display of Figure 4 
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graph he wishes to see, the graph begins at month 1 and 
then advances at one second intervals to the month 
requested. The user may also specify half-second intervals 
by a light pen detect. It is intended that this movement 
of the bar graph give a feeling for the history and current 
derivative of the statistic under scrutiny as well as the 
static numer’*s results of the simulation. In addition to 
the movement of the bar graph, the symbols >, and = 
are used on the display unit to indicate whether the value 
of a particular bar has increased, decreased or remained 
equal in comparison with the previous month. 

Figure 9 has no printed correspondent in this 
experiment. While the data for cost of lost sales, 
availability, total dollar investment and number of pur- 
chase actions generated are available as individual bar 
graphs as represented by Figure 8, all four of these are 
available at the same time in the Quadrant Graph. All 
move simultaneously under the same program timer control 
as the individual graphs. Each graph is scaled uniquely 
and, of course, represents the same data as are available 
on the individual graphs, but with the obvious reduction 
in resolution since each line of a graph has only approx- 
imately half the width of the display device screen to 
represent the data as does the individual graph. 

Figure 10 depicts the Program Function Keyboard (PFK) 
which is the means by which the user indicates to the 


computer what he wishes to see on the display device. 
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a5 
Key 0 will always display the user instructions if the 


plastic overlay, with no sense switch cut-outs, ® is in 
place. Otherwise, key O will return control to the 
zsraphics monitor. 

There is a two-step procedure for obtaining a 
display: 

1. Indicate which month is to be seen. 

2. Indicate the display data desired. 

In order to prevent inadvertent display of data 
beyond the month desired, month change was set up as a two- 
step action. First, key 30 must be depressed (at which 
time a tone signal is heard), then the key corresponding 
to the month is depressed. This key is lighted by program 
control as a reminder to the user of his simulated month of 
furthest advance. At this time, no change is noted on the 
display device screen. When a key corresponding to a 
display is depressed, the display then changes. Keys 1-12 
will display Figure 6 for policies 1-12 for the month 
indicated. Keys 13-24 will be ignored by the program at 
this point. Note that keys 1-24 signify different events 
depending on whether they are immediately preceded by a 
key 30 depression or not. Key 25 will produce Figure 7, 
keys 26-29 will produce Figure 8 for Availability, Cost of 
Lost Sales, Purchase Actions Generated, and Total Dollar 


Investment respectively. Key 31 will produce Figure 9. 


Stam, Component Description, pp. 30-32. 
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The user is allowed to refer to any past month's data. He 
may not, however, move ahead in simulated time by more than 


one month at a time. 





IV. THE SUBJECTS OF THESE X Peel 


The experiment was designed to examine the decision-~ 
making processes of experienced, practical and interested 
professional administrators of inventory control, to find 
out if graphical display devices do, in fact, assist the 
decision-making process. The subjects were solicited in 
such a way as to insure a professional audience. The 
Short course was designed as a graduate-level course which 
would attract only those who were interested in the subject 
matter and who were prepared to understand the material. 
The simulation model was the experimental vehicle and was 
used in examples and problems which were integrated into 
the course. The participants came to the course in statis- 
tical inventory control for their own professional advance- 
ment, under the auspices of the firms for which they worked. 
The participants were not informed of the experiment which 
was being conducted, and they received the full measure of 
instruction for which they came. The control of the human 
factors in the experiment proceeded quite smoothly with 
nothing to indicate bias or data contaminated by the fact 
that an experiment was in progress. 

The individuals who comprised the experimental 
group were solicited from nearby major manufacturing 


concerns specifically for the short course in statistical 
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inventory control. It was requested that candidates for 
the course have experience in inventory control and be in 
a decision-making position in the company. No more than 
two candidates were to be sponsored by any one company. 
The following firms sent people to the course: 

Burlington Hosiery, Burlington 

Burlington House Division, Burlington 

Cardinal Products, Durham 

Coca Cola, Durham 

Golden Belt Manufacturing Company, Durham 

International Business Machines, Raleigh 

Kayser-Roth Hosiery, Burlington 

Liggett and Meyers, Durham 

Sears Roebuck, Greensboro 

Sperry Rand, Durham 

Virginia Mills, Swepsonville 

Westinghouse Electric, Raleigh 

The participants attended the course between the 
hours of 9:00 and 11:00 A.M. weekdays for a two-week 
period. The morning hours were chosen specifically to 
insure that the attendees would have the active concur- 
rence and support of their sponsoring firms and would be 
scheduled for this absence from their regular duties. 

After a random division into two groups of nine 
individuals each, the following statistics applied to the 


individuals: 





BN, 











Presently 
in 

Years Experience Automated Inven- Decision- 

in Inventory WOlny “CONvmel Making 
Individual Control Exper lenec Capacity 
GiaOuDp of 
A 20 No Yes 
B LES No Yes 
C 20 No Yes 
D We Yes Yes 
E i No No 
F 5 Yes » Yes 
G 10 Yes Yes 
H 1 Mes Yes 
i Z No Yes 
Group) Lt 

J 5 Yes Yes 
K 1 No No 
Je 5 Yes Yes 
M 0 No No 
N Ze No res 
6) 20 Yes Yes 
P 6 Yes Yes 
Q Z No Yes 
R Zz No Yes 


After a substantial part of the course material had 


been given, the participants were given either a problem 
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book with twenty-four months of simulation output (Figures 
2, 3, and 4) or assigned to work a problem on the display 
unit at individual laboratory sessions. The second problem 
was assigned later ina similar manner except the groups 
now used the presentation method that they had not used on 
the first problem. It was explained that congestion on the 
display unit precluded everyone working both problems in 
this way. There was no attempt made to identify individ- 
uals by group membership or emphasize this distinction. 

The participants were not graded on these problems 
and were encouraged to work them as part of the education 
process of the course to investigate some of the funda- 
mental properties of the inventory control policies. 
Although all of the inventory control policies simulated 
for the two problems were discussed in class, the partici- 
Pants were not given the correspondence te identify 
specific policies by number on the problem to avoid obvious 
bias. The participants, then, were asked to make their 
judgment based solely on the evidence without subjecting 
them to the prejudice of prejudgment. The two problems 
were of equivalent difficulty, the only distinction being 
the order in which they were presented, 

An operational environment was explained for the 
problem. Higher level management had presented the parti- 
cipant with the output data for the set of twelve inventory 
control policies, and had requested a recommended ranking 


so that an implementation of policies could be decided 
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4) 
upon. The participant was encouraged by management to 
present his recommended ranking as soon as possible, but 
cautioned that implementation of the recommended policies 
could have serious repercussions in the firm, so his best 
professional judgment was required. Figure 11 was the 
work sheet to be used. Participants were told to ignore 
the first two items in the top line as they had no rele- 
vance in this problem (Group #, USING:). Participants were 
reminded to keep accurate figures on elapsed time spent 
studying each month's data, and were also reminded to mark 
the decision month at which they would have presented that 
ranking as a recommendation to higher level management, 
prior to completing the rest of the forn. 

In this experiment there were two possible ap- 
proaches to the problem of evaluating the quality of the 
decisions made. The first and most commonly used approach 
is to establish and furnish the subjects with the criteria 
of excellence which is to be used in evaluating the 
decisions. For example, in many cases the subjects are 
first given instruction, then tested to ascertain how well 
they perform as measured against the pre-established 
ericveria. 

This method of measurement may be dependable when 
the material is objective and the criteria are easily 
established. In inventory control (and many other prob- 
lems) the weightings to be used in reaching a decision are 


highly individual. For example, one firm may emphasize 
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Fig, ll.--Data collection format 





43 
high availability, whereas another may give heavy weight 
to low investment. When the experimental subjects are 
experienced in the field and have developed their own 
criteria for decisions, the method of measuring against 
instructor-set criteria is unsuitable. One cannot know the 
extent to which the subjects followed the instructor's 
criteria and the extent to which they followed their own. 

A distinguishing feature of this experiment is that 
it took an alternative approach to the problem of evaluat- 
ing the quality of decisions. The subjects were well 
acclimated to the decision process by experience, and were 
familiar with and interested in the substance of the 
course. Therefore, they were given no weightings, no 
criteria of excellence by the instructor. Instead they 
were explicitly told to apply their own several diverse 
sets of criteria. The quality of their decisions was then 
measured by the consistency of each subject's own ranking 
at the month of commitment with the later ranking when full 
information was available. 

Although this second approach diminishes the mean- 
ingfulness of comparisons which show how well the subjects 
agree with each other, such measures were taken as a matter 
of interest. The use of self-consistency rather than 
artificial criteria does, however, improve the credibility 
of the other statistics. It also substantially improves 
the generalizability of the results, for it shows the ef- 


fect on decision makers when deciding by their own 
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standards, rather than by those dictated by a simplified 
theoretical model. In substance, we have separated the 
decision process from the classroom environment and 


inserted it into an operational environment. 
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Two main analyses were performed on the data: an 
analysis of variance of the Latin square design, and a 
computation of rank order statistics. We will first turn 
our attention to the Latin square design. 

The concept of a generalized Latin square design is 
traditionally introduced by explaining a checkerboard 
pattern of equalearea plots of land used in agricultural 
experiments. If we are investigating the effect of soil 
treatment on the yield of a crop, the plots are chosen at 
random in such a way that every row and every column is a 
complete replication, which is to say that each treatment 
will occur once and only once in each row and each column 
on the checkerboard. In an experiment, this design tends 
to avoid any systematic variations in effects, such as local 
soil variations, which are not due to the treatment. 

In the Latin square design of this experiment, it 
should be noted that instead of experimenting on different 
Subjects in each cell of the square, the same experiment 
group was involved in both cells of a row. In effect, 
each group acted as its own control group. Experiments in 
which the same subjects are used under all q treatments 
require q observations on each subject, and are called 


experiments with repeated measures. In this experiment 
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q = 2 since we are dealing with two treatments and each 
subject is observed twice. This variation on the standard 
Latin square is described in detail on the following pages. 

Tne formal design proceeds from a specification of 
the model to an analysis of variance and hence of the 
Significance of observed differences. In the analysis of 
variance, the sum of squares (SS) is computed for rows, 
columns and treatments. From this, a variance estimate, or 
mean square (MS), is calculated for rows, columns and 
treatments. This is accomplished by dividing each sum of 
squares by the degrees of freedom (df) corresponding to 
that sum of squares. From here an F test? is used to find 
the significance level of the variation under investigation. 
We will be most concerned with the significance of any 


variance due to treatment. 


Group I 
(9 Individuals) Display Printer 
Group II 
(9 individuals) Printer Display 


Fig. 12.--Latin square design 


The 2 x 2 Latin square with repeated measures!° 


PWilliam L. Hays, oSotvatistics for ts yecnoLOogicius 
(New York: Holt, Rinehart and Winston, S53), pp. 368-369. 
10z, J. Winer, Statistical Principles in Experimental 
Design (New York: McGraw-Hill Book Company, 1962), 
Pp. 33 
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of Figure l2 was used to analyze the data for: 


IS 


Simulated calendar time to reach a decision 
(decision month), 

Actual elapsed time to complete the rankings 
through the decision month, 

Actual elapsed time to complete the problem 
after the decision month, 

Correlation coefricients for tne conststcic, som 


the committed decision with the final ranking, 


in order to produce an analysis of variance of these 


Sea ilsStics. 


The particular design was chosen in part to isolate 


that variation due to natural differences between the 


experimental groups. AS can be seen below, the separation 


of the source of variation due to group differences allows 


a better measure of that variation due to the form of data 


presentation, which is the statistic of real interest. An 


appropriate model for this 2 x 2 Latin square design tes" 
E (xX ) = + + TT 


In [1] , “is the grand mean, a. represents the 


effects associated with groups, /T 1 (,) the effects 


associated with subjects within groups, & ; the effects 


due to the order in which each subject did the two exper- 


iments and /3 the effects due to the form of data 


presentation. The analysis and expected values of the mean 
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squares are shown in Figure 13, for the 2 x 2 case of the 
p x p Latin square with n observations in each cell. Here 
A and B represent the factors associated with the & and/@ 
ertecuse 


Schematically the experiment may be represented by: 





al a 
“1 oa oe 
Co db, by 
Variation Source df — ae E (MS 
Between Subjects np-1 el 
Groups [= eee Ll ac +poe + np Ge 
Subjects within 2 P) 
Groups p(n-1) = 16 Ve + PGy 
Within Subjects np(p-1) = 18 
A (Order) p= 1 cee + Np e 
B (Treatment) p-l = 1 a + np ape 
Error (Within) p(n-1)(p-1) = 16 oF. 


Fig. 13.--Analysis of the 2 x 2 Latin square design 


The computational procedure for the analysis of 
variance involves a straightforward partition of the total 
sum of squares.l2 For this particular Latin square design, 
the computation is as follows: 

Correction Factor G*/np* ae) 


Total SS ee 3] 


lépays, Statistics for Psychologists, pp. 356-375. 
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order ~a“/np Lu 
Treatment Z.B“/np ee 
Cell Total (AB, ,)°/n 16] 
Paired Observations BPE) /P (7] 
Row £.ce/np [8] 
ss, = L4} - [2] [9] 
$8, = [5] - [2] [10] 
SSeeoioe = Lol eal at 
SoeUol eoure = eae oe sla 
eS error itn ot ee (3) 


The summations in all cases are over the whole experiment. 
Pa(k) represents the sum of the q observations on person nm 
in group k. Gis the grand total; A, B, C, represent the 
sum of the np observations for their corresponding sources 
of variation. 

The ratio for the F test is computed for each 
source of variation in the usual manner by taking the ratio 
of two mean square values of interest. F(u,v) will desig- 
nate the F value where u is the number of degrees of 
freedom in the numerator of the ratio and v the number of 
degrees of freedom in the denominator. F(1,16) then for 
this experiment is: 


F(1,16) = Noe eerie 


F(1,16) = MS)/MS . ror (within) 


7 MS 
Ih Lp aken, Lisanne subj w.groups 
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In order to test the error term for homogeneity of 
variance, [12] is also computed so as to provide the parts 
that may be checked by an ane test.?? This is acconm- 
plished by summing only over Group I and over Group II 
rather than over the whole experiment. In this way the twc 
components so derived for f13] are tested at F(8,8) for an 
appropriate significance level. The magnitude of this F 
ratio indicates the extent to which the observed data con- 
form to the model, and thus provides a partial check on the 
appropriateness of the model. 

Since the values of the decision month are positive 
integers, a skewed distribution was suspected and found to 
be present. The nature of the decision is also such that 
higher variances would be expected to be associated with 
higher means. The values were, therefore, transformed 
following a standard procedure by taking the square root, 
to make the variances more homogeneous and at the same 
mame tO normalize the saIawio It is this square 
root value which is used in the analysis of variance 
computations, and reported in the tables. 

It is of interest to compare the rank order of the 
inventory control policies at the decision month and the 


rank order at the twenty-fourth month for each individual. 


This comparison gives a measure of each individual's 
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consistency between the ranking decision made at the 
decision month and that ranking decision made when full 
information wasavailable. These two rankings for each 
participant are compared by means of a Spearman rank cor- 


15 


relation coefficient test. The computational procedure 


is as) follows: 


pence — eae ee 6 ZDS ) [14] 


S 
N (Nf-1) 


where D, is the difference in ranking for the jth inventory 
control policy, and N = lz, the number of inventory control 
policies observed. Once computed, these values for re. are 
treated as data for analysis in the Latin square design 
previously discussed. 

Since the r, values are known to have a skewed 
distribution, a transformation is made on each of the r, 
values in order to place the data on symmetric scale, so 
as to normalize the distrinueiene it is these trans- 
formed values for r, which are used in the analysis of 
variance computations. The transformation is: 


) ieeal 


1+r 
5 iti el = 





Ts transformed ~ 
1l-re 


This transformation is a routine procedure used on cor- 


relation coefficients and is justified and discussed in 


l5yays, Statistics for Psychologists, pp. 643-646. 


16p , G. DwwSteel.and J. oH. lormic, springs plesuana 
Procedures of Statistics (New York: McGraw-Hill Book 


Company, 1960), pp. 188-189. 
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detail im the ret erence. 

In order to find the vextenv Sto whlch rie memnbercmen 
each group ranked the policies the same way under the same 
conditions, Kendall's W coefficient of concordancet? was 
computed for each cell of the Latin square. With this 
Single statistic computed for each of the four cells, some 
comments can be made concerning the homogeneity of de- 
cisions under the conditions of each cell. The computation 


for Kendall's W statistic is as follows: 


A 72 
- 125 . 3(N+1) [16] 
m2N(N2=1) N-1 


where for this experiment m = 9, the number of individuals 
doing the ranking; N = 12, the number of policies ranked; 
and T; is the sum of the nine individual rankings given the 
jth policy (i.e., the sum of column j in Tables 9 through 
i. The statistic W is based on the extent of variability 
among the respective sums of the ranks. The basic rela- 


Eronship is: 


ee variance of rank sums 
maximum possible variance 


of rank sums 
which reduces to [16 | - From this it can be seen that W 
indicates the degree of concordance, or how well the group 


members agreed among themselves as to the rankings. 


Consider another statistic. For each of the 


l?vays, Statistics for Psychologists, pp. 656-658. 
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(3) = 36 pairs of subjects in a cell, calculate the 
Spearman coefficient for their two policy rankings at the 
decision month. The average of these coefficients is a 
measure of the agreement among subjects. This average may 


be computed directly from Kendall's W statistic by: 


mwW-1l 
average - = sae 


ma the interpretation of the result of this computation it 
should be noted that this experimental group was not 
attempting to apply a common criterion of excellence in 
making their decisions. Individual best judgment and 
experience in the decision process guided the problem 
solution. The course which the subjects attended neither 
taught nor encouraged standards in decision making or in 
performance goals. 

The difference between the two values of re just 
discussed is that the single value of average Le derived 
by way of Kendall's W statistic is a measure of concordance 
among all nine participants within a cell, whereas the 
individual values of re derived from Spearman's test area 


measure of each person's consistency. 
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The data analyzed and the results of the analysis 
appear in the following tables. The results were computed 
by PL/I routines. using single precision floating point 
arithmetic. Where the tables appear in the form of the 
Latin square, each cell is represented as in Figure 12. 
The order of the participants in the listings is constant 
over the tables presented; that is, the i*) deta element 
always refers to the it) individual regardless of which 
table is referenced. The within-cell mean and standard 
deviation, corrected for sample size, are included in the 
data lists. 

In the analysis of variance we are examining the 


two treatment hypotheses: 


Ho 114, = /45 
Hy 1/4, >A 5 


That is, the null hypothesis is that the mean associated 
with the display unit is equal to the mean associated with 
the printer output. Since we believe that the display 
unit is, in fact, the better decision-making tool of the 
two, the alternative hypothesis is that the mean associated 
with the printer output is strictly greater than the mean 


associated with the display unit. We are also comment- 
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ing on the two order hypotheses: 


Ho 1 A451 =f oo 
Ay tM > Mo 


involving the means associated with Order 1 and Order 2. 
The usual method in statistical hypotheses testing involves 
setting the significance level of the test in advance of 
obtaining the data. The convention used in the analysis of 
variance deviates somewhat from this formality. The value 
of F is reported to exceed a specified percentile by a 
comparison with tables of critical values. The choice of 
the significance level is in this way determined partly by 
the data itseir. 1° This allows each reader to establish 
his own significance level and to judge the results there- 
by. We will consider a conclusion to be more surely 
established if the probability of its truth is higher. 
This methodology does not allow a measure of the power of 
the test; however, the procedure is valid for estimating 
the probabilities of the observation in relationship to the 
assumed sampling distribution. in this preliminary inves- 
tigation we would be encouraged to find significance at 
the .05 level, and the significance levels actually 
ercoauned are.noted in the tables. 

In this analysis of variance, we are principally 


interested in investigating the effect of the treatment on 


the subjects. We will also be interested in commenting on 


18yiner, SLauIstlcal Principles, Pam. 
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tne effect of order on Che wsubiects. “ine wrestles wollen 
correspond to these effects will be referred to as the 
within-subject sources of variation. We have also reported 
the effect which corresponds to differences between Group I 
and Group II as the between-subject source of variation. 
The results of the various analysis of variance computations 
are shown in Tables 2, 4, 6, and &. 

In the analysis of variance for the ordinal value of 
the decision month, we were compelled to use a trans- 
formation to produce a more symmetric distribution of 
values, as noted in Chapter V. Both the original values 
for this month and the transformed values are presented 
in Tables la and lb, respectively. These values represent 
how many months of data the participants needed prior to 
committing themselves to a decision on the rank ordering 
that they would recommend. The question is: How early in 
simulated time can a decision be made? 

The null hypothesis that the variance is equal over 
the groups cannot be rejected, as the ace test indicates 
(Table 2). The significance level of .10 appears to be 
marginal, and we cannot really say that we accept the null 
hypothesis either. However, we are sufficiently satisfied 
with this result to have confidence in the validity of the 
analysis of variance for the treatment factor. For the 
treatment factor, Ho is not supported at the .01 level. 

We can say with a high degree of assurance that the mean 


decision month arrived at by using the display unit was 
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TABLE la 


ORDINAL VALUE OF DECISION MONTH 


Order 1 Order 2 

13 14 

6 14 

8 BES 

10 Display al Printer 
Group lI 12 mean=10.222 17 meaneie3.222 

10 G = 4,086 14 U'= 2.048 

19 14 

8 Le 

6 10 

9 8 

S, fe 

10 9 

8 Printer 8 Display 
Group II 13 mean= 9.333 10 mean= 8.111 

8 GT = 2.121 7 GQ = 1.269 

6 fi 

eZ 7 

9 IU) 


indeed less than that using the printer output. From 
another viewpoint, the participants on the average needed 
to look at less data volume with the display unit to make 
a committing decision than they did with the printer output. 
There is only a 1 percent probability that these apeeuued 
differences are due to chance effects. No effect can be 
attributed to order, as indicated by the significance level 
pum c > «fOr order. 

Reference to Table la gives some additional infor- 
mation on the characteristics of the data within each cell. 
Within each group, the mean decision month is lower for the 


presentation on the display unit. The standard deviation 
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TA SEE kb 


TRANSFORMED ORDINAL VALUE OF DECISION MONTH 


Order 1 Order 2 
Sracleh: Sel 
2.449 36741 
ae Fae 3.605 
3.162 Display 3.316 Printer 
Group I 3.464 mean=3.145 4.123 mean=3.626 
Se alloys G = .607 3.741 D= .282 
4.358 Bie ee 
24026 3.464 
2 A449 Belo? 
3.000 CROeo 
3. OG 22045 
She 2 3000 
2.025 Printer ZeOeG Display 
Group II 3.605 mean=3.037 3.162 mean=2.840 
= = 2.828 Be his 2.645 G-= 4256 
2449 2.645 
3.464 2.6045 
S000 3.162 


does not follow this pattern however. For Group I, the 
standard deviation, @ , of values for the display unit 
presentation, 4.086, is considerably higher than for the 
printer output, 2.048, while just the opposite is true for 
Group II (1.269 vs. 2.121). In other words, both groups 
had a higher G@ for Order 1 than they had for Order 2. 
Apparently there was some maturation between Order 1] and 
Order 2 which was reflected in @, but not in the mean in 
poth groups. The participants who were presented with 
Order 1 on the display unit were encountering both a unique 
problem and a unique way of analyzing the data, whereas 


Group II presented with Order 1 used a familiar data 
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TABLE 2 


ANALYSIS OF TRANSFORMED DECISION MONTH 


Signifi- 
ait F(1,16) cant at 


Variation Source 


Between Subjects 





Groups aI ROS 
Subjects w/in 

Groups 16 
Within Subjects 
Order 1 “25 
Treatment 1 Om 
Error (Within) 16 

Homogeneity of Variance, x rest: 


Signifi- 
Variation Source ee) cant at 


Within Subjects 
Error (Within) Group I 1.148 8 
Error (Within) Group II 398 8 Aaa 


ae 
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presentation for data analysis. Note that one very 
cautious participant, who waited until the nineteenth 
month to commit himself to a ranking, did quite a bit of 
damage to the standard deviation for the Group I, Order l 
cell. 

The elapsed clock time which was required for a 
participant to commit himself to a ranking is given in 
Table 3, and its analysis of variance in Table 4. The null 
hypothesis of homogeneity of variance cannot be rejected by 
the Fmax test, allowing us to accept the results of the 
analysis of variance. 

The treatment factor is shown to be significant at 
much beyond the .001 level. The mean time to decision 
using the display unit is with great certainty less than 
that using the printer output. This indicates that the 
amount of time spent in making a decision was significantly 
reduced in this experiment by using the display device, 
which may point to an ability to assimilate a large quan- 
tity of data and to correlate these data by using display 
techniques, as opposed to printer output. The convenience 
of having virtually instantaneous recall of data displays 
by using the Programmed Function Keyboard is certainly a 
consideration in the interpretation of the results. Push- 
ing buttons is just inherently faster than paging through 
a book of data, however well arranged and indexed the book 
may be. However much or little this consideration entered 


into the results, we still can give credence to the 
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TABLE 3 


ELAPSED TIME IN MINUTES TO DECISION MONTH 








Order 1] Order 2 

63 67 

a) 85 

75 1ay 

68 Display 72 Printer 
Group I 88 mean=64.666 115 mean=89.777 

79 G-=13.500 70 0 =23.156 

81 114 

pe 89 

al 69 

52 4] 

69 29 

60 17 

66 Printer Bal Display 
Group II 109 mean=73.444 43 mean=39.222 

61 0=19.255 148 T= 6.457 

95 38 

61 Bi 

88 39 


proposition that data can be correlated faster and retained 
better from a properly programmed display unit. 

Unsolicited comment from individual participants 
supported this conjecture without exception. The obser- 
vation of the author is that the dynamic graphs gave the 
participant a much better intuitive feel for the situation, 
and that he was more likely to retain this impression and 
not have to refer to past data repeatedly. One consider- 
ation in this observation was the program control which 
always started the dynamic graphs at month 1] and brought 
them up to the current month in increments of one second. 


This forced a continual review of the history and 
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TABLE 4 


ANALYSIS OF DECISION TIME 


Signifi- 
Variation Source Ss cha MS F(1,16) cant at 


Between Subjects 


Groups 
Subjects w/in 





3927.125]| 1 3927.125| 9.161 Od) 


Groups 63582125] 16 
Within Subjects 
Order 186.750] 1 not 
Treatment 7921.60 0Gal £2.00] 


Error (Within) 3469.250| 16 





Homogeneity of Variance, Fyay Test: 


Signifi- 
Variation Source SS af Ja iS few, cant at 


Within Subjects 
Error (Within) Group I 1806.43 8 
Error (Within) Group II 1662.78 8 1.086 not 
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derivative of the measure under consideration and undoubt- 
edly reinforced past impressions. It was seldom that a 
participant asked that the graph be stopped at a month 
prior to his current month so that he could review the 
static situation as of that past instant. As a matter of 
interest, it was noted when the experiment was well under 
way that the more experienced participants used the dynamic 
graphs extensively, where the less experienced participants 
relied on the tabular listings presented on the display 
sug biee 

Referring to Table 3, we note that both the mean 
time to make this decision and the standard deviation with- 
in the cell are considerably reduced in favor of the 
display unit over the printer output. Group I, in going 
from display unit presentation to printer output presen- 
tation, showed a mean increase of about twenty-five minutes 
to make a decision, and an increase in Q@. Group II, in 
going from printer output to display unit presentation, 
showed a mean decrease of about thirty-four minutes and a 
very sharp drop in © . It might be argued that in spite 
of the maturation between Order 1] and Order 2, Group I did 
better on the display unit, and that the very noticeable 
difference between Order 1 and Order 2 statistics for 
Group II is due only in part to this maturation. It is 
interesting to note that only one participant (Participant 
F) made a faster decision using the printer output than 


using the display unit. If we examine the data in terms 
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of order, it can be seen that within each order the display 
device gave more favorable results in terms of mean and 
standard deviation @Ghan did the printer output. 

The statistics of Tables 5 and 5 for the total 
elapsed time to complete the problem after the decision 
month, and its analysis of variance, are included for 
interest. It can reasonably be assumed that there was some 
speed-up by participants after they had made their decision 
to commit, simply to get to the end to see how well they 
did. There is some evidence of this speed-up in the time 
data. How much of this is due to increased familiarity with 
the problem at hand and how much is due to impatience to get 
to the final result is difficult to say. The elapsed time 
after the decision month was tempered by the requirement 
that the data be ranked at each month. From personal 
observation, the participants appeared to be conscientious 
about following the spirit and the letter of the instruc- 
tions, but relieved that the big decision had been reached, 
and were ina hurry to finish the twenty-four months to 
check their final ranking against their decision month 
ranking. 

Table 6 shows that the hypothesis of homogeneous 
variance cannot be rejected, and we can go directly to 
comment on the within-subject sources of variation. Here 
the order statistics show that Ho cannot be supported at 
the .01 level, which is sufficient to support a statement 


that we accept H,}. The order factor is thus seen to show 
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TABLE 5 


TIME IN MINUTES TO COMPLETE THE PROBLEM 
AFTER THE DECISION MONTH 








Order 1 Order 2 
62 4y 
7a 47 
87 54 
64 Display 55 Printer 
Group I 50 mean=59.777 28 mean=44,.555 
64, 7 =17.788 32 Tagen 
fell 70 
56 3D 
63 39 
58 OM 
36 51 
83 51S. 
65 Printer 52 Display 
Group II 68 mean=72.666 54 mean=51.000 
106 Q=22.237 LG J=11.135 
93 66 
54 5D) 
91 25 


significance here, wnich is reasonable and consistent with 
our previous comments on the effects of order. In this 
test the treatment factor was completely without signifi- 
cance which is also a reasonable result in view of the 
observation concerning the impatience to finish the problem. 
Reference to Table 5 Shows a completely order-dependent 
data structure. 

In Table 8, the analysis of variance statistics for 
the transformed Spearman rank correlation coefficient, Yr.,, 
are presented. The Fy, test is satisfactory in our 


investigation of equal variances, so we turn our attention 
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TABLE 6 


ANALYSIS OF TIME TO COMPLETE PROBLEM 
AFTER DECISION MONTH 


Signifi- 
Variation Source SS df MS ee late), cant at 


Between Subjects 


Groups 841.000 841.000 sO 
Subjects w/in 

Groups 3466 .000 AMS lof § 
Within Subjects 
Order 3061.750 P61 
Treatment 93.437 not 
Error (Within) Be 3 Le 





Homogeneity of Variance, IO ear Test: 


Signifi-e 

Variation Source Sic af MElarn ier cant at 
8 
8 


Within Subjects 
Error (Within) Group I 2 leas 


Error (Within) Group II 2918.00 eyo not 


Towne [usp 
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to the within-subject sources of variation. Here our 


hypotheses must be modified to read: 


Treatment Factor Order Factor 
Hi: Mp>m> t'§MoosAo1 


The hypothesis of equal means for the treatment factor, Ho, 
cannot be supported at the .05 level. With a significance 
level of .05, we obviously do not have the clear mandate 
meMmav our other treatment factor statistics have given wepus 
we are willing to claim that the evidence points towards 
eyeccLlon Of Ho and acceptance of Ay that is the mean 
correlation coefficient is higher using the display unit 
than using the printer output. The values of Ops give a 
correspondence between the participant's ranking of the 
policies at the decision month and his ranking at the last 
month of the simulation data, month 24. This, then, is a 
measure of the consistency between these two rankings. It 
is also a measure of the participants discrimination abil- 
ity - that is his ability to decide whether he has enough 
information to commit himself to a ranking or not. A 
decision to commit too soon in relation to each individual's 
ability and ranking criteria would, in most cases, result 
in a poor correlation coefficient, whereas being overly 
cautious and waiting beyond the point where he had suffi- 
cient information could not be expected to materially 
improve the correlation coefficient. Thus, we might say 


that one interpretation of a low re would be that the 
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participant committed himself too early. Other interpre- 
tations are, of course, that he simply used poor judgment 
in his ranking, or that he materially changed his ranking 
criteria after the decision month. Participants were 
cautioned to use a consistent ranking schema throughout. 

As an extreme example, it was pointed out to the class that 
to rank the policies based only on lost sales data through 
the decision month, then to abandon that schema and to rank 
the policies only on number of purchase orders generated 
would not be showing responsible judgment. On the basis of 
these comments, we should be able to narrow our consider- 
ation of a principal cause of low r, to either too early 
commitment or poor ranking judgment at some point. Both 
of these essentially are measures of decision-making 
ability and we can accept either one or both as reasonable 
interpretations of a low correlation coefficient. 

From Table 8, we can see that the order factor is 
highly significant; the probability that the mean of 
Order 1 is equal to the mean of Order 2 is something less 
than 0.1 percent. Thus we accept the alternative 
hypothesis, Hy» that the mean of Order 2 is greater than 
the mean of Order 1. The maturation between Order 1 and 
Order 2 would appear to play a large role in the decision 
making consistency that is being measured. That is, the 
ability to meaningfully rank a set of policies grows with 
practice, and moreover, improves more than the actual time 


and simulation decision month considerations previously 





TABLE 7a 


SPEARMAN RANK CORRELATION COEFFICIENT r,. 











Order 1 Order 2 
2615 a 
A405 oa 
soll ale: 2685 
412 Display ° 559 Printer 
Group I 447 mean=.501 965 mean=.647 
vA =.195 700 O55 
510s: VDE 
» 7 34 BO55 
475 Bas. 
-~.062 .678 
~ 398 PAL 
-.580 Lokal 
0279 Printer AV iTA® Display 
Group II .209 mean=.126 720 mean=.760 
2188 C=, 302 £923 =.073 
405 VEE 
»209 a7 o3 
~090 ROO 


measured. Reference to Table 7a shows that the mean for 
both groups increased from Order 1 to Order 2, but that the 
standard deviation for both groups under the display unit 
treatment was less than the standard deviation for both 
groups under the printer output presentation. The data for 
the standard deviation in this table are of the same 
general pattern as the data for the standard deviation in 
Table 3, wvbut Table 7a is the only table examined so far in 
which the means within both groups increased with order. 
One prominent difference in this comparison is that 

Group II performed very poorly under the printer output 


for Order 1 and quite well under the display output for 





70 


TABLE 7b 


TRANSFORMED SPEARMAN RANK CORRELATION COEFFICIENT rg, 





Order l Order 2 
One 2e7 0.985 
0.430 reig95 
0,119 0.839 
ae eer | er Printer 
Group I O.481 mean=. 59 Z2Ok mean=.901 
O.cl3 G"=,266 0.879 =. 524 
0.893 0,621 
0.937 0.056 
Oey 0.893 
-0.063 0.825 
0.421 0.953 
-0.663 W530 
OyzZo7 Printer 0.906 Display 
Group ITI 0.212 mean=.124 0.908 mean=1.030 
0.191 Vie 1.609 = .236 
OeeLe TeOoo 
0.091 0.879 


Order 2. The lowest and highest means and standard 
deviations for all four cells are found in the Group II 
data. The use of printer output for Group II had the 
highest standard deviation, and the lowest mean, while the 
use of the display device for this group had the highest 
mean and the lowest standard deviation. Interpretation of 
the data from Table 7a in conjunction with the treatment 
factor data from Table 8 gives more credibility to the 
acceptance of Hy for the treatment factor. 

Tables 9 through 12 show the rankings of each policy 
by the participants at the decision month. Each of the 


tables represents one cell of the Latin square. For 
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TABLE 8 


ANALYSIS OF TRANSFORMED SPEARMAN RANK 
CORRELATION COEFFICIENT, rg 


Signifi- 
Variation Source ss lie MS PCI, low cant at 


Between Subjects 


Groups 
Subjects w/in 
Groups 


<2 


Within Subjects 
Order 


Treatment 
Error (Within) 


4.001 
AOS. 





Homogeneity of Variance, x rest: 


Signifi- 
Variation Source F(8,8) cant at 


Within Subjects 
Error (Within) Group I 1.854 
Error (Within) Group II e723 2.563 


7s a 





Ve 
example, in Table 9, Participant A ranked policy 1 as 
eleventh and policy 2 as seventh at the decision month. To 
demonstrate the correspondence among the tables, reference 
to Table la and Table 3 shows that Participant A reached 
this ranking at month 13 after an elapsed time of sixty- 
three minutes. For Tables 9 through 12, Kendall's W 
statistic and more importantly, the derived value of 
average re is indicated. The values of average r_ from 


S 


these tables are summarized as follows in our usual format: 





Order l Crder 2 
Display Printer 
Group I neers Ss) r= 664 
Printer Display 
Group II r=e 543 r=-829 


As was the case in Table 7a, the effect of order is evident 


here from the increase in average r. from Order 1 to 


Ss 
Order 2 in both groups, and the most prominent increase is 
in the movement from printer to display unit output in 
Peoupelia, Within thesconsideratiionget order, seme cei 
which represents the display unit data had the hisher 
value of r, for both Order 1 and Order 2, This would tend 
to indicate for this experiment that whatever the order, 

a more homogeneous ranking was made by the participants 


under display unit presentation than under printer output 


presentation. 





TABLE 9 


DATA AND RESULTS FROM KENDALL'S W STATISTIC 
COMPUTATIONS FOR UPPER LEFT QUADRANT 
OF LATIN SQUARE; GROUP I, 
ORDER I) Dis r EA aon 


Policy Number 


ae 1 2. 3h 67 ee oO 
A 11 7 6 2 106 9512 eee 
B 6 2° 7 & 10 5 12008 9c 
C 7 3° 8 19S 65 ie eee 
D 8 9% 9G A 10S ieee eee 
E 12 10 2 3 68 2) -5e se eee 
F 3 1 8 5 9 6 M2 11 7 
G AS ail 1 12) “oa Omens 
H 9- 2 8 1,9 -6 129117) 72 ecu 
i 9 2 6 & 109 12 11" Sy cue 


Summary of ranking by participants at decision month; 
W = .621 


average r= O74 
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Comet er Ef IN: tA ew 
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TABLE 10 


DATA AND RESULTS FROM KENDALL'S W STATISTIC 
COMPUTATIONS FOR UPPER RIGHT QUADRANT 
OF LATIN SQUARE; GROUP I, ORDER 2, 
PRINTER Our Pur 


Policy Number 


a” 12 3 4 5 6 % 8 0 Tor 
A 9 5 12 7 11 10 1 3 4 Geom 
B 6 2 12 8 9 7 2 38 5 ome 
C ? & 12 8 10 9 T°WiBP Secu 
D 6. 7? 5 8 21 10 2 Oe) 1 eee 
E 6 1 12-9 7 10 3°92) (Seance 
F 3 2 12 9-98 5) teed 7 ilo 
@ 6 3 12 § 8 10 2. 0107 ioe 
H 10 9 6 11 8 3 92 1 7 ae 
I 7 3 6 8 Tl 10 t 3 5) oie ee 


Summary of ranking by participants at decision month; 
W = s7UL 


average rT, = .664 
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Ai: ey Ula aka 


DATA AND RESULTS FROM KENDALL'S W STATISTIC 
COMPUTATIONS FOR LOWER LEFT QUADRANT 
OF LATIN SQUARE; ScnOUP IT OnDER Ll 
PRINTER OVMeUT 


Policy Number 


ae 1 2 3 4°53 96 79s o eo 
y 9 5 6 1 7 8 Wt "12 2 ice 
K 6 1 9 2 10 3 It 120055) Soe, 
iL 7 3 2 12 dt & 6 10 (Oe Eee 
M ef 46 § 7 §8 12 10) oui wee 
N 6 1 7 3 10 & 12 12 5 Ceo 
O hoo4 6 3 9 9 12 16 96 es iene 
P 7 & 9 3 10 6 1221 oe eee 
Q bw 9.2 8°95 711 92°) ioc mene 
R 7 -@ “9? 1 "10. 6 1a) 2? "Scueee e 


Summary of ranking by participants at decision month; 
Wo o= .593 


average—ars =e 43 





TABLE 2 


DATA AND RESULTS FROM KENDALL'S W STATISTIC 
COMPUTATIONS FOR LOWER RIGHT QUADRANT 
OF LATIN OOo Gh Olle ie 
ORDER M24) Blo PLATO T 


Policy Number 


Sgaheertant Ze Gua 5 NG = 7 Ol ilo 
j o 3 6 8 10 it ears oneay 
K 7 9 9 1 12 0a Coe 
C 6 2 7 6 1) 100.) sire 
M 6 3° o. 10 eee ee 
N 6 OE os. lt So re 
O 3. 2 6 2 Ge ii eee ee ee 
P Se ie > ME oe 2 pe 7 
Q 5 2 6 8s 0 7 ee ee ee 
R Gwe 3 Sizes 11 10) Sea eee 


Summary of ranking by participants at decision month; 


W = .848 


average T_ = ~829 
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It should be emphasized that the value of average r, 
derived by way of Kendall's W statistic (Tables 9 through 
12) is a measure of concordance among all nine participants 
within a cell, whereas the values of ie derived from 
Spearman's test (Table 7a) are a measure of each person's 
consistency. It is also worthwhile commenting again that 
the concordance was not measured against predetermined 
criteria, but rather reflects the degree to which experi- 
enced decision makers tended to agree under different 
treatments but using their own best judgment. 

Tables 13 through 16 give the results of the 
decision process at the end of the twenty-fourth month 
when the decider had full information available to him. 
These tables represent the data in the same manner as did 
Tables 9 through 12. That is, in Table 13, Participant E 
ranked policy 1 as last, policy 2 as tenth and policy 3 as 
iiemch., The resulting average r= is represented ingeur 


usual Latin square format as: 


Display Printer 
Group I 639 ~856 

Prenver Display 
Group II ABS 947 


Here the pattern is much the same as it was for the 


decision month average r. (see page 72 ), except that the 
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TABLE 13 


DATA AND RESULTS FROM KENDALL®S W STATISTIC 
COMPUTATIONS FOR UPPER LEFT QUADRANT 
OF LATIN SQUARE AT TWENTY-FOURTH 
MONTH SGCR OUP Uren brn: 4 
DISPLAY UNIT 


Policy Nounter 


Partici- 
pant 1 2 3 by S 6 7 8 S 10 i 
A 12 6 7 9 10 8 5. eal al Ly fi 
B 8 cE) Ese 2 7 5 9 10 5 6 Z 
C le 5: JO ae 6 9 i, 8 1 3 2 
D EZ > oe a 7 4 S510 fa 6 1 
, le Ae 9 4 (a ish 5 6 3 8 1 
iS 6 Hl Ce eez ae al 9 3 v 2 
G i oH 12°20 . oan 6 2 9 1 
H az oy led 4 6 So 0 9 2 7 al 
ib 9 1 el 2 i 6 8 10 ZC 4 3 


Summary of ranking by participants at twenty-fourth 
month: 


W = .679 


average Yr. = 639 


MO WwW F FF WW 


Ny SA 
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TABLE 14 


DATA AND RESULTS FROM KENDALL‘'S W STATISTIC 
COMPUTATIONS FOR UPPER RIGHT QUADRANT 
OF LATIN SQUARE AT TWENTY-FOURTH 
MONG GROUP 1, OnUER 2 
PRINTER. OUTER 


Policy Number 


Partici- 
pant i} 2 3 Ly = 6 7 8 9- 10 Sia 
A 7 2 ee 8 & i2 3 ue eae) 9 Ly 
B 6 2 ed 9 C ee 8 1 4 12 8 5 
C 7 Zea, 9 S33 3 I: Saale 8 Ly 
D i 1 10 9 6 1a 3 i2 pie p iby 8 4 
Ez 6 fa 88) 9 (ogy eal 5 aL 4 12 e) 5 
F 4 2 C02 6 eis ik Oe 9 4 
G (4 2 Ga ae 9 3 5 Ly Sit 1 
H 7 2 9 6 4 12 > aL 3- Lees 8 
i 6 pie 0, 9 (aa 3 Z 4 12 8 5 


Summary of ranking by participants at twenty-fourth 
month: 


W = .872 


average rf, = WBNS 
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TABLE 15 


DATA AND RESULTS FROM KENDALL‘S W STATISTIC 
COMPUTATIONS FOR LOWER LEFT QUADRANT 
OF LATIN SQUARE AT TWENTY-FOURTH 
MONTES GROUP it Onan 1. 
ede lbs. OSE ARE 


Policy Number 


Partici- 


pant 


J 
K 
L 


M 


@ 


WM © dW 


t 2 3 4 5 6 7. *89) 9.10 Sine 
12 8 2) 5 G10 6) GC eee 
16, 2 2s 1 | 7 ec eo 4 3 «5 
12 21 99 5 89) 10 ee ee 

G6 5 3 12 12 16. “O01 7 cee 
125 211 10 6 7 8) 96:3 eee 

B 2 9 12 11 10 7° oo) S30 

9 1 112 %.°5 8 10 92 ee 

8 1.11 12 9 §5-°9 9165) 3a 
12° Sie 10 5 ~9 6 7 eo ee 


Summary of ranking by participants at twenty-fourth 


month: 


W 542 


average T. = A85 
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Telbig lhe 


DATA AND RESULTS Fign nEN CTE SW ors hie 
COMPUTATIONS FOR LOWER RIGHT QUADRANT 
OF LATIN SQUARE AT TWENTY-FOURTH 
MONDE eaeGh OUP SieiaG ep Emeez 
DISPLAY UNIT 


Policy Number 





Partici- wens 
pant 1 2 30) gS oe ono ee 
J (f 4 10 9 Cuil Z 1 Sy ell 8 3 
K 6 Zoe) 9 (ane | 3 1 ume 8 Ly 
L 6 ele 9 (ea 3 at Sel 8 Ly 
M D 1 9 8 (ei 4 Z i eee eee £6) 6 
N t) fg deal 9 60 3 1 Sai 8 4 
O 4 YZ 9 8 oe a 5 1 i lee C 
i 6 0 9 (fe Ala 3 @ 4 12 8 5 
Q > Zyl 8 7 9 3 A Hi 2 ee 6 
R 6 eee 9 7 ee 3 2 4 12 8 5 


Summary of ranking by participants at twenty~fourth 
month: 


We NSS 


average Yo = .947 
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effect of Group II] going from printer output av Order @ 
to display unit output at Order 2 is much more pronounced. 
Phere was a moderate increase in the average r. for Group I 
zoing from display to printer output, which may be ascribed 
in part to maturation. However, the average r, almost 
@oubling in Group II when going from printer ouvsueF to 
display output may be more than can plausibly be ascribed 
to maturation alone. In comparing these results with the 


average r. results from Tables 9 through 12, it would be 


S 
well to remember that the rankings from those tables did 
not represent the rankings made all at the same month. In 
the case of the rankings at the twenty-fourth month, the 
values of average reg may be a more valid measure of con- 
cordance since the decision relating to the ranking is not 
confused with the additional decision of when to indicate 
a decision month. In the rankings at the twenty-fourth 
month, the deciders all had the same amount of information 


available to them, which was not the case for the decision 


month rankings. 
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point to the use of a display presentation when economy of 
time or simply volume of printed output is a serious 
constraint on the system or the decision maker, 

While the two results reported above have the 
rigorous respectability of high statistical significance, 
the next results to be discussed are at least as important 
in the evaluation of the experiment. These are the results 
which answer the question of whether a better decision can 
be made with a display device. Because of the criteria 
used, we will claim that a decision at the month of commit- 
ment that is more consistent with the final decision is a 
better decision, and also that a decision which exhibits 
evidence of higher concordance among group members is a 
better decision. However, the two major results listed 
above would be convincing in their own right even if the 
decisions turned out to be no worse using a display device. 

The results from Kendall’s W statistic and from 
Spearman's rank correlation statistic show that individuals 
tended to make more consistent decisions with themselves 
and with their group using display device presentation 
techniques. Casual inspection indicates the conclusions 
established by the statistical analysis; within order, 
individuals made more consistent decisions using the 
display. It is equally clear that within group the 
increase in correlation in going from printer output to 
display output (Group II) far exceeds the increase in 


eommelataon in @oingefrom displaymeutput togprinter outpouL 
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(Group I) for both of these statistics. 

The evidence from the Spearman and the Kendall tests 
definitely indicate better decision making under display 
device presentation. With this result we conclude the 
evaluation of the statistical results. 

The economics of a system of display devices for 
decision making will not be explored here, however, it is 
evident that the very specialized research equipment used 
for this experiment is both expensive and unnecessarily 
elaborate for an operational system. The prices and 
characteristics of display equipment that will serve the 
purpose very well are frequently published. 1? 

The minimum display unit for implementation of an 
information system of this general type should have alpha- 
numeric display capabilities and a programmed function 
keyboard, or equivalent means of easy display selection. 
The size of the display face is crucial to the extent that 
it must be able to contain enough material to be of inter- 
est and still allow character size and spacing to enhance 
readability. For instance, the IBM 2250 used in this 
experiment has a display face twelve inches square with a 
maximum capability of fifty-two lines of seventy-four 
characters each. The information in the displays (Figures 
6, 7, 8, 9) is rather densely packed and is digestable 
only by someone sitting in the console chair immediately 


19adams Associates, Computer Characteristics 
Quarterly, Second Quarter 1968 (Watertown, Massachusetts: 


eydata and Adams Associates, Inc., 1968), pp. 179-187. 
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in front of the display face. A smaller display face 
would mean that displays would have to be segmented; the 
same information in smaller characters on a smaller display 
face would be the wrong compromise. With segmented 
displays, more programmed function keys would be needed and 
in some ways the problem of how to ask for a particular 
display becomes more complicated for the user. It is 
unfortunate that the great majority of the pure alpha- 
numeric display units have small display faces - eight 
inches by six inches appears to be a popular size. Other 
features of the IBM 2250, such as an alphanumeric type- 
writer keyboard, line-drawing capability, and a light pen, 
would serve only to detract from the simplicity and utility 
of the system and add only to the cost of this application. 

In addition to the display device proper, this 
experiment used other system facilities. The display unit 
had a self-contained buffer of 8,192 characters. Of this, 
a maximum of 2,000 characters of puffer storage were used 
at any one time. The display program in the main storage 
of the IBM 360 Model 40 used approximately 13,000 characters 
for program and 21,000 characters for tables. An additional 
46,000 characters of disk storage were used for table 
overlays. 

How generalizable are the results of this experiment? 
The display system evidently achieved the objective of 
presenting a complex situation, which involved many inter- 


relationships, ina manner such that the key concepts and 
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fundamental correlations were clearly understandable. The 
display system appeared to facilitate interpretation and 
extrapolation of the relevent data; a significant element 
in the decision process. The reduction of reaction time of 
top-level decision makers in this environment is both an 
interesting result and an important objective of any 
executive display system. While there is an obligation to 
report these results in relation to the experiment group 
itself, the composition of this group allows us to extend 
the results to a wider group of individuals. It would 
appear reasonable to apply the results to that class of 
individuals who are experienced in and actively making 
decisions in the inventory control field. It is tempting 
to extend these results to decision making in general; 
however, that extension will have to await further investi- 
gation and experimentation. 

What direction should further work take? One point 
of great interest would be the exploration of the 
differential cost or savings of decisions using display 
units and printer output. This is a rather difficult area 
to define in that dollar values and weightings must be 
assigned not only to such elements as a reduction in 
inventory valuation and the cost of lost sales, but also to 
the generated purchase actions, availability of material, 
timeliness of decision, system cost and other influencing 
factors. The advantage of having such a differential cost 


at hand is obvious; just how to go about obtaining it is not. 





as 
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Statistics on the frequency of use of the various 
displays should be collected, both automatically and by 
experimenter observation. The correlation of the frequency 
of use by display type with the individual’s consistency of 
decision would be most important for the design of exten- 
sions of this system. Unfortunately, the importance of 
this statistic was not realized until this experiment was 
well under way. 

Whatever the extension of the program, there should 
be the capability for the decider to request a hard copy of 
any display he wishes. If line drawing capability is used, 
this, of course, implies the availability of the equivalent 
of a line plotter for hard copy output. This requirement 
is more operational than experimental. We have no doubt as 
to the utility of such a feature for the decider in an 
operational environment, and if a display unit has a line 
drawing capability, it should be used with this requirement 


in mind. e 
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